JSON schema conversion benchmarks - #196

✅ Deploy Preview for schemabenchmarks ready!

Name	Link
🔨 Latest commit	e03d690
🔍 Latest deploy log	https://app.netlify.com/projects/schemabenchmarks/deploys/6a78a91660eba4000872f15a
😎 Deploy Preview	https://deploy-preview-196--schemabenchmarks.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

Looks good. Thanks for picking it up. I think the only comment that support matrix on the home page would be more relevant and interesting to see than performance of draft-20.

Also, the order of home content doesn't follow the sidebar, which I found misleading.

@EskiMojo14 Hello!

AJV and ATA currently show as supporting both 2020-12 and Draft-7 ... but TB is only showing 2020-12

https://deploy-preview-196--schemabenchmarks.netlify.app/

Reference:

Here is the specification coverage implemented by TB:

https://github.com/sinclairzx81/typebox#coverage

Here is TB type-level support for Draft 7 and Draft 2020-12:

TypeScript Reference Here

Question:

Does it make sense to include AJV, ATA, and TB in a benchmark specific to JSON Schema conversion? It isn't really an apples-to-apples comparison, and they appear to be visually penalized for not having JSON Schema conversion functions, even though they all intrinsically support the specification, which seems a bit misleading.

Would it not be better to narrow participation to libraries that actually provide X > JSON Schema conversion?

In my understanding, it makes sense to include all libraries, because these are 2 different operations:

• Create a schema/validator from JSON Schema - this is where TB, AJV, ATA have good coverage
• Create JSON Schema or Standard JSON Schema from schema - this is where TB and others are penalized, because they use JSON Schema for internal implementation, which doesn't convert to all standards out of the box.

Both of them are important and have different use cases. So I think ideally there should be two separate benchmarks From JSON Schema and To JSON Schema.

I also think that the PR, as it is, is already good, and this should go as a follow-up PR.

I also think that the JSON Schema coverage number and TypeScript inference support are good cases for the follow-up PR.

To summarize, to address the confusion @sinclairzx81 highlighted, I'd rename "conversion" to "To JSON Schema".

And then add a separate PR for "From JSON Schema" because this is a separate thing. I can also help create a foundation for it.

for JSON schema based libraries, we already measure "from JSON schema" - that's what the initialization benchmark is. there are a few libraries that support creating schemas from JSON without being based on it, which might be interesting to benchmark.

I can see the argument that it doesn't make sense to measure "conversion" for libraries that use JSON schema as their base. I think the cleanest solution is to remove them from this - the compliance benchmark (#198) will be a more useful "support matrix" than this provides, anyway.

I'm not entirely sure what the best approach for Typia in particular would be - they don't use JSON schema as a base, but the "conversion" is still in build time.

okay, lots of changes later, here's a summary:

• renamed existing benchmarks to conversion.toJson, and removed non-runtime conversions.
• added conversion.fromJson benchmarks (maybe missing some)
• support matrix removed in favour of compliance testing (will merge both together)

thoughts appreciated :)

Looks good! I like the split.

1. Still think that performance is not really relevant for the conversion cases and should be secondary, because it's usually called only once at process startup. But it's not critical at all.
2. Compliance testing is good, but it makes toJSONSchema dependent on the library's fromJSONSchema ability. I think it'd be better to have a simple support matrix for toJSONSchema, including Standard Schema support, and then a separate compliance matrix + roundtrip for the fromJSONSchema case.

Nothing blocking, just my opinion.

Although, as you said, the support matrix can be merged into compliance, just for libraries not supporting fromJSONSchema, it'll be an unknown JSON Schema compatibility number. Need to experiment whether 1 or 2 tables will look better/cleaner.

that's a fair point - the compliance testing is a support matrix for from JSON, but not for to JSON. I'll have a think on how best to present that information - like you said, having two tables made the page feel overcrowded.

okay - added support matrix back, and added tabs to the page so only one table is shown at a given time. i also created a card version of the support matrix for mobile users.

Hi, I like how the UI looks. I just have a few comments regarding data; I hope I don't overwhelm you 😅

• ajv and ata-validators don't make sense for to json schema matrix - they are not schemas, but validators. You create a JSON schema from a validator, but you can't create a JSON schema from a validator.
• From the other side I think TypeBox should be included, since it's not a validator, but pretty much a full-fledged schema - the fact that it uses JSON Schema as internal representation is an implementation detail. But I believe it should only be in the Draft7 section, and the other ones should be marked as +- or something like this, since there's no automatic conversion, only if the user specifies it during schema creation.
• I think it would be nice to have an extra column for toJSONSchema support matrix showing Standard JSON Schema support with native|native-opt-in|additional-package options - the table looks like a nice place for it
• Sury supports S.fromJSONSchema 😁

Ah, and forgot to mention one more. I think the matrix on the home page would be a better overview